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Remarks 

The Drawings filed on July 21, 2003 are objected to by the Examiner. Figure 
1 of the Drawings is amended. A replacement sheet showing the amendment 
to Figure 1 and a request for drawing amendment are included with this 
response. The Specification is amended to include missing reference 
characters in the description. 

Claims 1-20 are pending in the application. Claims 1-20 are rejected. All 
rejections are respectfully traversed. 

The invention provides a virtual reality environment by acquiring 
concurrently, with a plurality of cameras, a plurality of sequences of input 
images of a 3D object, each camera having a different pose. The plurality of 
sequences of images is reduced to a differential stream of 3D operators and 
associated operands. A 3D model of point samples representing the 3D 
object from the differential stream is maintained. Each point sample of the 
3D model has 3D coordinates and intensity information. Then the 3D model 
is rendered as a sequence of output images of the 3D object from an 
arbitrary point of view while acquiring and reducing the plurality of 
sequences of images and maintaining the 3D model in real-time. 

Before the rejection is traversed in detail, Applicants would like the 
Examiner to take note that in computer graphics two types of model 
representations are commonly used. One model uses a polygons, usually 
triangles, and those models can be called a triangle meshes. The triangles or 
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polygons are two-dimensional surfaces. The other model is a point model, 
which uses 3D discrete points in space. The number of references that 
distinguish these models are too numerous to cite, but any standard reference 
on computer graphics will suffice. The Applicants respectfully request the 
Examiner to understand and respect this difference, as reflected in the 
Manual of Classification. Those of ordinary skill in the art would never 
confuse point models as claimed and triangle models as cited by the 
Examiner. 

The primary references cited by the Examiner only describe triangle models. 
Applicants assert vigorously now, and will continue to do so in the future, 
that the applied art is irrelevant and unrelated to what is claimed. Applicants 
further request that in the future, if the Examiner is going to consider art, it 
will be art related to point sample models, and not triangle models. 

Claims 1-4, 7, 8, 10 and 13-18 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Matusik et al. ("Polyhedral Visual Hulls for Real-Time 
Rendering" - Matusik) in view of Deering (U.S. Patent No. 5,867,167). 

The Examiner states that Matusik reduces sequences of images to a 
differential stream: 

System, 1 st paragraph); reducing the plurality of sequences of images to a differential 
stream (page 123, Section 4 Real-Time System, 1 st paragraph); rendering the 3D model 
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With all due respect, reducing is not described by Matusik. Instead, Matusik 
describes background subtraction, a stream of silhouettes and texture 
information: 

the video stream at 15 tps and performs the following processing steps: First, it seg- 
ments out the foreground object using background subtraction. Then, the silhouette 
and texture information are compressed and sent over a lOOMb/s network to a central 
server. The system typically processes video at 320 x 240 resolution. It can optionally 

Silhouette and texture are both data. Silhouettes represent shape, or 
'geometry' as it typically referred to in computer graphics. Texture maps 
represent indirectly surface intensities. There are no operands in Matusik's 
stream. 

Matusik does not describe a 3D model of point samples from the differential 
stream. In contrast, Matusik's model only represents the silhouette (shape) 
data: 

We assume that each silhouette s is specified by a set of convex or non-convex 2D 
polygons. These polygons can have holes. Each polygon consists of a set of edges 

In constructing his model, Matusik does not consider differential 3D 
operators and operands, and certainly not intensities. 

Matusik does not render a 3D point model. 

Moreover, Matusik teaches away from the invention, see title "Polyhedral 
Visual Hulls . . ." The Matusik model is constructed of 2D polygons, (also 
known as triangle meshes, or polyhedral hulls, see title), not 3D discrete 
points. Matusik makes it quite clear that for his model it is an advantage to 
reduce to the silhouettes to 2D data structures in section 2.2: 
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2.2 Reduction to 2D 

The intersection of a face of a cone with other cones is a 3D operation (these are 
polygon-polyhedron intersections). It was observed by [10, 16] that these intersections 
can be reduced to simpler intersections in 2D. Tins is because each of the silhouette 
cones has a fixed scaled cross-section: that is, it is defined by a 2D silhouette. Reduc- 
tion to 2D also allows for less complex 2D data structures to accelerate the intersections. 

In the prior art, 3D point (particle) models and 2D polyhedral (triangle 
mesh) models are clearly distinguished, e.g., see US 6,396,496, Pfister, 
"Method for modeling graphical objects represented as surface elements." 
Also see the most well known standard reference text on computer graphics, 
J. Foley, et al., "Computer Graphics: Principles and Practice," 2nd ed., 
Addison- Wesley, 1990. 

There is no doubt in the art of computer graphics that point and triangle 
models are essentially incompatible. In fact, a considerable amount of 
processing needs to be done to convert one model to the other. Neither 
Matusik, nor Deering below, which also operates on triangle meshes 
describe such conversions that could make them applicable to what is 
claimed here. 

Furthermore, the Matusik polygon model or "visual hull" only represents 
object shape. There is no intensity information in the Matusik visual hull. 
Intensity-like information is stored in texture maps, not the visual hull. 

Deering does not reduce input images to a differential stream of operators 
and operands. Deering delta encodes the geometry of his triangle model, see 
column 12: 
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described. Many techniques are known for minimally rep- 
resenting variable-length bit fields but for the geometry 
compression according to the present invention, a variation 
2D of a conventional Huffman algorithm is used. 

and at column 17: 

... ..... 

Opcode within state machine 220 processes the values ~ 
provided by the Huffman tables 230 and outputs data to the 
barrel shifter unit 240. State machine 220 also provides an 
output to data path controller 260, which outputs a prefer- 
ably 12-bit wide signal to a tag decoder unit 294 and also 
outputs data to the barrel shifter unit 240 and to a normal 40 
nrocessor 270. and a nositinn/color nrocc^snr 2K0. 



Note, in Deering, the model is generated in step 200, and the delta encoding 
of the triangle model in steps 230 and 240 takes place later. Deering does not 
encode a sequence of images. 

Also, there are no operators in Deering. 

Furthermore, in Deering, the source generates graphic data, not a sequence 
of images. Therefore, Deering cannot be combined with Matusik. Again, 
those of ordinary skill of the art would realize that Deering uses a triangle 
model, and not a point sample model. 

Matusik does not describe transmitting a differential stream from the first 
node to the second node by a network. 

Matusik does not represent a moving object by a 3D point model. 
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Matusik does not represent a segmented object by a differential stream. 

Applicant cannot find insert, delete, and update operators in a differential 
stream anywhere in Deering. The Examiner generally references columns in 
Deering. The Applicants respectfully assert that the words "insert", 
"update", and "delete" do not appear in Deering. Clarification as to which 
terms in Deering are believed to be their equivalents is respectfully 
requested from the Examiner. 

Matusik does not have point samples. 

Matusik has a single shape/geometry model for all cameras. Section 3 in 
Matusik deals with texture rendering. As stated previously, texture and 
shape are treated separately in Matusik. 

Matusik does not have point samples. The rendering in Matusik maps 
textures to triangle mesh, see Matusik throughout. 

Matusik does not have a differential stream. 

The normals in Deering are surface normals of triangles, and not normals of 

point samples, see column 9: 

The compression of surface normals will now be 50 
described. Traditionally 96-bit normals (three 32-bit IEEE 
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Reflectance properties of point samples are not described by Matusik or 
Deering. 

Claims 9, 12, 19 and 20 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Matusik et al. ("Polyhedral Visual Hulls for Real-Time 
Rendering") in view of Deering (U.S. Patent No. 5,867,167) and further in 
view of Pauly et al. ("Spectral Processing of Point-Sampled Geometry"). 

As stated previously, point sampled models are incompatible with triangles 
models. None of the art of Pauley can be combined with either Deering 
and/or Matusik. Specifically point splatting techniques can never be applied 
to triangles. 

Claims 5 and 1 1 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Matusik et al. ("Polyhedral Visual Hulls for Real-Time Rendering") in 
view of Deering (U.S. Patent No. 5,867,167) and further in view of Kanade 
et al. ("Virtualized Reality: Constructing Virtual Worlds from Real 
Scenes"). 

Kanade also uses a triangle mesh for his model. None of the art described by 
Kanade can be used to reject what is claimed. 

Claim 6 is rejected under 35 U.S.C. 103(a) as being unpatentable over 
Matusik et al. ("Polyhedral Visual Hulls for Real-Time Rendering") in view 
of Deering (U.S. Patent No. 5,867,167) and further in view of Lee et al. 
(U.S. Patent No. 5,684,887). 
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Lee deals with recovering background using monocular vision. There are 
two basic problems with Lee. First, the invention discards the background; 
therefore, Lee is of no use. Matusik also discards the background; therefore, 
Lee cannot be combined with Matusik. Furthermore, Lee deals with 
monocular vision. In the claimed invention, the cameras are at different 
poses, and by definition can not be from the same view. The art of Lee is 
useless. 

It is believed that this application is now in condition for allowance. A 
notice to this effect is respectfully requested. Should further questions arise 
concerning this application, the Examiner is invited to call Applicant's agent 
at the number listed below. Please charge any shortage in fees due in 
connection with the filing of this paper to Deposit Account 50-0749 . 



Respectfully submitted, 

Mitsubishi Electric Research Laboratories, Inc. 




Clifton D. Mueller 
Agent for the Assignee 
Reg. No. 57,836 



201 Broadway, 8 ,n Floor 
Cambridge, MA 02139 



Telephone: (617) 621-7517 
Customer No. 022199 
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